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INTRODUCTION 

The purpose of this paper is to describe a human factors evaluation of the MH-53J helicopter cockpit. 
This evaluation was an application and further development of Tools for Automated Knowledge 
Engineering (TAKE). TAKE is used to acquire and analyze knowledge from domain experts (aircrew 
members, system designers, maintenance personnel, human factors engineers, or others). TAKE was 
successfully utilized for the purpose of recommending improvements for the man-machine interfaces 
(MMI) in the MH-53 J cockpit. 


METHODOLOGY 

One of the most difficult tasks facing designers, programmers, and engineers is creating the knowledge 
base to define system requirements. Once the tedious and perplexing task of eliciting the knowledge from 
experts is complete, the resultant mountain of data and the analysis and distillation of this data often 
becomes a major choke point in the design process. TAKE alleviates these problems by applying a 
formalized methodology to acquire knowledge from experts, and providing tools to facilitate the analysis 
of that knowledge. 

TAKE utilizes concept mapping to capture information collected from domain experts (DEs). Concept 
mapping was proven to be a superb knowledge acquisition technique for design during the development 
of the Advanced Knowledge and Design Acquisition Methodology (AKADAM) (McNeese, Zaff, Peio, 
Snyder, Duncan, McFarren, 1990). TAKE was created from the ideas generated and the knowledge 
gained while utilizing AKADAM. TAKE applies concept mapping to create integrated knowledge bases 
that are then analyzed with computer assistance to distill and extract information that allows the human 
factors engineer to develop a sound man-machine interface. 


Concept Mapping 

Concept maps are verbal-graphical knowledge representations created during an interactive interview 
process in which the domain expert and the interviewer share a common information construct. While 
the DE is expounding on the topic of interest, the interviewer creates the concept map on a drawing 
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board representing his or her interpretation of the relayed information. The DE can actually see how the 
interviewer is interpreting his or her comments and descriptions, and can alter and correct the concept 
map as it is constructed. Iterative inteiviews and reviews of the map further enhance the information 
content of the structure and ensure its accuracy. A unique characteristic of the concept map is t at it is 
virtually unbounded in the types of relationships that it can convey. In addition, concept maps can be 

created from notes, or text. 


Computer Analysis 

TAKE software transforms the information content of the maps into a data base that can be categorized 
and formatted to facilitate analysis. Two of the software tools that were utilized extensively for this 
evaluation are an Outline function and an Analysis function. 

The Outline tool does just what the label implies - it organizes the map content into a hierarchical outline 
format that improves readability and groups information into logical clusters. This function has proven 
extremely useful in generating functional decompositions and in organizing large scale data bases n 
addition, the outline can be edited and the changes will be reflected m the concept map. The ! out me 
feature is very powerful because it produces a universally recognized format that can be utilized by 
multiple disciplines. This universality makes it possible for users, engineers, designers, and programmers 
to fully understand the domain of interest. 

The Analysis tool allows categorical information to be selectively extracted from the maps using a 
"graphical spreadsheet" interface. Several categories of key terms can be defined with this function and 
are typically grouped in functional and descriptive classes. Verbal constructs within the concept map t at 
contain any of the defined key terms can be identified, extracted, and examined along with related 
concepts. With this function, information can be easily categorized for problem identification, task 
assignments, or other characterization. Categories can be color coded and displayed on the concept map 
to provide a visual representation of the data. 

Another very important feature of TAKE is the Drawing tool. The Drawing tool significantly improves 
the speed of entering concept maps into the computer format. A unique feature of the drawing tool 
enables the user to embed information within a concept map. Submaps, text, and graphics can be 
embedded in nodes to provide addition information on particular concepts of the map. ma icons are 
depicted in the nodes that contain this embedded information. Clicking on these icons opens windows 
that display the embedded information. Consequently, a map can be extremely rich with data and contain 
a great amount of detail but still maintain an uncluttered overall view of the domain. In addition, t e 
drawing tool provides the user with the ability to move, delete, or zoom portions of the map, an 
highlight nodes of interest by size, shape, or bold lines. 

Other functions TAKE include a combining feature that allows the integration of data bases from 
multiple maps and an alphabetical sorting function for listing all the verbal constructs within the maps. 


RESULTS 

A request for a human factors evaluation of the MH-53J PAVE LOW helicopter provided an excellent 
opportunity to test the TAKE process. 
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To conduct the evaluation, several methods were used to gain an understanding of the cockpit, the 
mission, and how the design of the cockpit impacted the safe and successful performance of that mission. 
The methods used included reviewing Technical Orders, Operators Manuals and checklists. In addition, 
traditional techniques such as interviews, questionnaires, and observations made during actual flights 
were used to determine mission requirements and cockpit configuration. 


Interviews 

During the interview process, traditional interviews and concept maps were utilized to elicit knowledge 
from the domain experts. This allowed comparison of the two approaches. The anchor used during both 
sessions was a generic mission profile that was developed with experts on our initial visit. During both 
techniques, the DEs were asked to explain the things that they did during each segment of the mission 
profile. They were asked to be very specific and describe the tasks that they performed, whether they 
were focusing inside or outside the cockpit, what cues they were looking for, etc. 

The interviews and checklists were the basis of the task analysis. Constructing the task analysis from the 
traditional interviews, was a tedious process. First, audio tapes were reviewed to fill in the notes taken 
during the interviews. Then the information needed for the task analysis was extracted from each 
interview. These steps were performed for each individual interview. Constructing the task analysis with 
the concept maps was a much simpler process. First the maps were entered into TAKE and a database of 
the information was created. Outlines were then produced from the maps using the Outline feature 
previously described. The maps were then combined according to crew position. Pilot maps, copilot 
maps, and flight engineer maps were combined so that only one document per crew position had to be 
analyzed. This streamlined the process because one map was used instead of paging through several to 
obtain the same information. The outline format quickened the process of building a task analysis because 
it was easy to extract the functions, procedures and tasks from the indentations of the outlines. It is 
estimated that it took half the time to construct the task analysis utilizing TAKE versus the more 
traditional techniques. 


Questionnaires 

Questionnaires were distributed to aircrew members who were asked to rate a number of instruments, 
panels, or controls with respect to lighting, legibility, visibility, functional grouping, access, location, and 
utility. In addition, a section was provided for the crew members to make any additional comments that 
were not addressed in the questionnaire or that needed further explanation. Statistical Analysis Software 
(SAS) was use to analyze the numeric ratings and the TAKE was used to evaluate the users written 
comments. 

The Analysis tool was used to sort the data into five categories. The five categories used are listed below 
along with an example of some of the keywords used to identify that category: 

Spatial - reference to location, e g. "near," "overhead" 

Temporal - anything that referenced time, e g. "day," "night" 

Resources - requirements, e.g. "need," "require" 

Differences - negatives, e g. "couldn't," "didn't" 

Visual - anything that dealt with the visual processing of information, e g. "see," "look," "NVG" 
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A color coded map was produced which coded the five categories to provide a visual representation of 
the data. Based on this color coded map, it became very obvious that the bulk of the problems were first 
visual, then spatial. This was extremely useful because it focused the analysis process. Until this point, a 
definite approach had not been decided because of the volume of information that was involved. This 
categorization of the data focused the evaluation and led the human factors engineers to find ways to 
improve visual information processing and access to various instruments in the cockpit. It was also 
interesting to note that the bulk of the spatial problems were identified by the flight engineer. This was 
not surprising because the observations and task analysis revealed that the flight engineer was required to 
access panels and instruments located all over the cockpit. 


Observations 

Observations included visits to the aircraft and actual flights. These observations provided a means to 
further identify problem areas and visualize what the operators were saying in the interviews and on the 
questionnaires. Also, cockpit measurements were taken for instances where crew members remarked that 
a certain instrument was hard to reach or hard to see. These measures were documented and compared 
with visual angle and anthropometric requirements. A concept map was created from the notes and 
measurements taken during the observations. Analysis was performed and again problem areas were 
identified as being mainly visual and spatial. 

An engineering drawing of a proposed cockpit modification was shown to operators after each interview 
session. They were asked to comment on what they liked and disliked about the design. They were also 
given the opportunity to re-configure the design using hard copy cut outs of the controls and displays. 
This allowed the operators to design their own cockpit and explain why they wanted certain controls and 
displays located in the various positions. 


SUMMARY 

The MH-53J helicopter evaluation has verified concept mapping to be a superb knowledge acquisition 
technique and has proven the TAKE process to be very effective in organizing and analyzing the acquired 
information. 

Concept maps collected during interviews proved to be an effective means of eliciting knowledge from 
DEs. They were also very useful in organizing user comments on the questionnaires and notes taken 
during observations. 

The Outline and Combine functions streamlined the process of building a task analysis from information 
organized in the form of concept maps. The Analysis tool made it very easy to extract the information 
needed from the observations to document in the report. In addition, the categorization of visual and 
spatial problem areas allowed the human factors engineers to focus on ways to improve visual 
information processing and access to various instruments in the cockpit. Based on the results of this 
evaluation using these methods, an optimal cockpit configuration was recommended for the MH-53J. 
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